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INTRODUCTION 

Educational testing has increasing^ come under public and professional 

scrutiny. One only has- to look atf emer^ifig social developments such as 

"truth in testing 11 legislation, recently passed in New York and now being 

considered by other state legislatures and the U.S. Congress, A National 

-Consortium on Testing comprising more than three, dozen'national educa- 
i 

tional organizations has also recently been established to review 
standardized, testing in detail. , The American Psychological Association 
(APA) , the American Educational Research Association (AERA\, and the 
National Council on Measurement in Education (NCME) have announced another 
joint committee to revise their "technical standards on testing"; this 
represents their third revision in the last two decades (Haney, 1978). 
Such developments have led one test developer to conclude that "the going 
is. getting tougher for educational testers" ( Boston Globe , 1978). 

In part, increased public 'and professional awareness 'tan be explained 
,hy tfce widespread use of these- tests in the -last two decades. The sale of 
standard tests had grown fivefold since the early sixties (Association of 
American 'Publishers, 1980), and now amo\ints to an annual sum o'f over 200 
million dollars (Kohn, 1975). This boom is due mainly to the growing 
use of tests by local and state educational agencies: At the local 
level, there is evidence that tests ^are commonly used to gauge the suc-~ 
cess'qf individual schools and the performance of ' teachers (National 
School Boards Association, 1977), as_we^L as to -identify and manage • 
individual student differences in the school and classroom (Leyine, 19'76) . 
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Other studies report an enormous increase in state-sponsored testing', 
both for determining school districts 1 advancement (Kembie, 1976) and 
for setting minimum competency standards for graduation (Haney & Madaus, 
1578). One study in California reported that 55 percent of the .schools 
made curriculum changes as a result of a recent statewide testing program. 
Others have pointed to the increase in testing at the state and local levels 
to meet- federally mandated or federally supported assessment activities 
(Clasby, 1973). It has even been suggested that standardized tests are 

now so thoroughly ingrained irr-American schools that "it is a sign. of 

* ti 

backwardness not to have test scores >in the school records of children 

(McClelland, 1973). 

One mainstream of concern about educational testing focuses on test 

makers — private testing firms. *This concern stems from a recognition 

that private testing firms, unlike other private organizations in education, 

are unregulated through formal governance arrangements. That is, they are 

* » 

held accountable^ neither to .<those elected nor to their representatives 

(Cohen, 1979). There is some truth in this view. Most other private 

'and quasi-private organizations—for example, professional associations 

(like the American Federation of teachers or the National Association of • 

Elementary 'School. Principals) and public interest groups (like the 

'Children's Defense Fund and' the National Association for the Advancement 

of Colored People)— are held accountable in their lobbying efforts by 

elected officials or their representatives at the federal,- state, and 

local levels. * This is evident in accounts of such formal public decision-' 

* * * 

making processes : as the U,S. Congressional legislation of categorical 
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programs (Bailey-,1965 ; Thomas, 1975), state legislative setting of school 

-curriculum (Marconnit, 1968; Bowles, 1966), and local school board hiring 

policies (Gilbert, 1966), These organizations are also^t&J^d^ac^t^tabl^ .by 

judges (appointed representatives of public officials) in their fcourt,cle- 

* » 
cisions on matters such as school finance reform, edueation of the handicapped. 

and desegregation.. While -views differ about the extent to which these 

processes act to hold private and quasi-private organizations accountable 

to the public,* private testing firms are not accountable through any 

fotmal means to elected officials or their appointees—though this does 

• ■ • « 

not necessarily .mean that they are not accountable to the public. 

One school of political-economic theory suggests that the public as 

consumers can hold private firms accountable through the workings of , the 

marketplace (Friedman, 1962; Dahl, 1953). By deciding what tests to buy— 

and not to buy-cc)nsumers> can be seen, as exercising their political 

preferences, just as others may exercise theirs by voting~~Ih elections. 

♦ 

Like elected public officials, private enterprise competes to meet public 
needs 3iid demands. Iif it fails?* to meet those tv^eds^and demands, consumers _ 
will not buy its products and it* will suffer from'a lack of public ^support:. 
Accqrding tq this view, then, school districts acting as consumers .can 



* For example; one school of 'political theory ^argues/that private and 
quasi-private organizations are self-appointed public representatives 
wbo have the advantage of competing with more* diffuse constituencies of 
elected officials and thereby can exercise a^ disproportipa&te influence 
over public decision making (see Madison/ 1961; Bendix & Lipset, 1967; 
McConnel, -1967; Kariel, 1961). Conversely, pluralistic political theory 
argues that these private influences are part of the democratic process, 
4s evidenced by the conflicts and compromises preceding consensus in 
.public decision making (see -Dahl 1953;* Truman , 1951; Lindblom, 1977). 
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hold private testing firms accountable' through the marketplace. 

Nevertheless, concern about the accountability of private testing 
firms persists. Some 'argue that because these firms are monopolistic 
and lack competition, they shojw little daring in the pro'duct they offer 
their consumers (Kohn-, 1977). In this view, they are portrayed as public 
utilities but without public control. A related concern is that private 
testing firms offer their consumers a unif orm'product that provides little 
opportunity for real or significant choice (Hoffman, 1962; Levine, 1976) — 
perhaps as one might see major car manufacturers' such as General Motors, 
Ford, and Chrysler, but without international competition. Others 
suggest that private testing firms have enough influence with government 
officials that few efforts are made to regulate or monitor their activi- 
ties (Karier, 1972, Kiersh, 1979). This view tesembles charges made 
against the food' industry despite the existence of a Federal Food and 
Drug Administration (Hunter, 1972). Others'still feel that private ' 
testing firms' interest in profit has eclipsed their responsibility for 
ensuring that their customers know how to use their products correctly^ 
(Oles, 1977) — perhaps 'like a toy manufacturer who sells a gun that can 
harm^ children if it is improperly loaded.' A. further charge is that pri- 
vate testing firms sell their product utider false pretenses — claiming that 
their test can provide information that it. cannot. For example, there is 
some question whether educational achievement tests do in fact measure 
achievement or are simply another meksure' of ability (Levine, 1976) 

While tTie list of concerns abcftit private testing firms 1 lack of - 
public accountability is long, there has been little formal study of the 

9 m • - \ 
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issues they raise. An important issue that cuts across many of these 
concerns is the extent to which private testing firms are in fact held 
accountable through ,the workings of the marketplace. Do. consumers influ- 
ence and control the tests produced by these firms, or do the latter 
fexeircise 'sovereign control over the tests they produce and sell? What 
attention has been paid t'o this question has been focused on admissions 
testing £or colleges and professional schools and on the main provider 
of such 'tests, Educational Testing Service (ETS)£e.g., Nairn, 1980; 
National Education Association, 1977; Strenio, 1980). ' Considerably less, 
attention has been given to this issue at the elementary and secondary 
•school level;' yet mast of the nation f s 17,000 (circa) school districts 
age reported to buy standardized norm-referenced educational^ achievement 
ifests from private testing firms (National School Board Association 
'"standardized Testing 11 Research Report, 1977). The extent to which, 
S9hool districts as test consumers influence private testing firms is 
particularly important to examine, because elected school officials or 
those reporting to them are invested — through statute or tradition — with 
authority over education. -Should they lack influence as consumers, this^ 
might indicate a weakening of local control and democratic responsibility 
and ap undue and unaccountable exercise of power by private testing firms. 

This essay focuses on the role of sphool districts in the marketplace. 
It will assume that school districts are rational consumers — that they will 
buy the test that meet£ their needs and wants, much as drivers buy parts 
for their c&r — with a specific need and purpose In mind. Thus, 'through 
the kinds of tests they buy, they will influence what private testing 
firms produce. But this assumption will also be examined, since to the 
extent that consumers do, not know what they want and need, their influence 



3 



in the marketplace may be .limited. For Example, schqol 'districts might 
buy tests as consumers buy girl scout cookies — with/ a sense of fulfilling 

a social and cultural obligation, and with lit tie/at tent ion to the product 

* , , . J >' • 

as such. In this case, their influence on private ^testing firms is likely 

to be limited. . To explore the, role of* loc^l school districts as consumers, 

then, this essay will examine* how *atid on what basis local school districts ^ 

select and. buy standardized norm-referenced educational achievement tests, ^ 

Answering that question* will have some important implications for assessing 

the effectiveness of the marketplace as a means of local school district ^ 

control and ^iHuence over the^tests they buy. The essay will conclude 

with a discussion of the marketplace versus' formal 'governance arrangements 

for meeting" local school distric^. testing needs, ^ 

HOW TESTS ARE BOUGHT 

A How local school districts buy. tests is an important question to 

answer because it illuminates their behavior as consumers. In particular,- 

it provides a way to assess whe'ther they buy what private testing firms 

persuade or tell them to buy, Lindblom (1977) calls this "circularity 11 

' in' the marketplace, because the needs and wants of consumers are determined- 

by those selling the product. A: classic example of circularity in the 

marketplace Is that of the door-to-door "salesman seliing a ."housewif e a 

bpish she does^not need-; her decision to select and buy a brush is based 

less on her own need than that 'of .the seller, % Similarly, if local school 

districts tend to r buy what they are told to buy, they can be assumed to 

be less than rational consumers and to have little influence in the 

marketplace — just as cbuld be expected in the case of the housewife,^ 
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In'order to, assess^ the extent to which private testing firms may 
enjoy this^ positio'n in the' marketplace, several preliminary questions 
about the behavior of school districts as confers need to be answered. 
These include why a school district selects a test in the first place", and 
what the relationship is, between school districts an'd-*private testing firms 
in the selection process. The following sections answer these questions by 
describing why and when a test is selected and who is involved" in the 
selection process-. a 

C 

Why and When a '/Test' is Selected ^ 

Teachers, principals, administrators, and School board members com- 
monly agree that thei* school district enters the market for standardized 
norm-referenced tests mainly for one reason: to inform their community on 
"how 'their schooUdistrict an'd^ schools are doing compared to the larger 
population;" (Interview with Director "of Public Services, 12/13/7$ )\ p. 1). 
School, officials typically regard the provision o^ such information as a 
necessary part of their accountability 'to the local taxpayer. And even 
though there are school districts that w6uld rather not subscribe Jo such' 
a means of; accountability, they recognize that there is little choice. As 
the director of research, in one school^district explained, if he did not 
do the testing "the community would-be suspijpious.'because other school 

districts do it" (interview, . 12/18/79, p.,l).>y' 

\ / ' ' < 

Several additional reasons encourage a school district to select 

.standardized norm-referenced tests.' For' example, some school officials 

see them as a helpful administrative tool',, say. for comparing student and 

staff performanc/'ln deferent schools and classrooms. . Others may $ee 

them as a way t6 assist in *the, improvement of instruction. * 
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For whatever reason 01; combination' of reasoti^school districts 
might be in-the marker for a standardized ^norm-referenced test, they ^ 



•typically change t^sts anywhere- f rom- every three to every ten year's. 

Kv ^ The decision to change tests is usually prompted by .teachers 1 and admin- 

• istrators ^dissatisfaction with__the tpe\t in current use. 'Nat atypically, * 

> aylchoolvofficial in one district explained that the superintendent had 

-set up a committee to select a new test because teachers had done 1 a ,ltit * 

of grumbling and complaining atiout the test" (Interview with Research ■ 
• , * \ • / ' • , ' - 

Coordinator, 2/1/80, p. 1). Such complaints may stem from the fact th^t 

. V ■ * ' 1 ■ 

the test* uspd has fallen behind changes in the curriculum (Interview • 
with. Assistant 'Superintendent for Instruction, 5/23/80, ty. 4), iLs "too 
< long" (Interview with Director of Special Services, 2/5/80, p. 1), or is 
* "not helpful, to instruction^ (Interview with teacher,^ 1/31/80, p. 3). , 
Despite ^dissatisfaction with a test in use, however, a new test is. 
normally selected only i£ someone influential in the district presses for 
it\ For example, a superintendent or head' of the testing program may think 
that there is "a better* test on the market" (Inperview* with Director of 
Pupil Services and. Special, Education,' 5/23/80, p. 2)— that is, that a new' 
test seems to address one or several complaints prevalent in the school 
district. .On some occasions, a new test may be selected because it is 

. '.. reputed to be "easier" and promises toValte a school district ^ok^ptter 

* * \ « . ' 

.' . id the -eyes t>f 'the community. For' example, a superintendent in^ small • 

rural school district explained that he had decided on a new test mainly 
k . i '* ' * • ■ * " • "\ ' ' 

. 'because his colleagues in another .school district had told him that their 

' . ' ' ' x 

"scores had gone up" (Interview, 12-13/^^^3). There was no evidence 

' across , school districts that this test uniformly produced that result; 



V 



ERLC 



12 



some school , districts , using it y found^ that their scores dropped. But y 
school officials tried to select Whichever test they thought would reflect 
favorably on their -school district. » r 

Choosing the standardized, test t#vbe used *by the school district*!* 
not always easy. The view tfcat a change should be ma^e^is* rarely shared 
by everyone in the district and is usually balanced against other jreasonsf 
for keeping the old test. These include the advantage of retaining the , 
uniform and longitudinal data base provided by 'long-term use of the sa$e 
test. For example, a guidance counselor spoke of his frustration when the 
board, failed *t& approve his proposal to change tests because a board member 
did want to lose the accumulated years of "baseline data'l (Interview, 

1 * \ 

5?l9/80, p. 1). Another common reason ior not changing tests. %*s given by 

a ' ■ 

a director of research. Though aware that- the test, in use was 4t out of 

date, 11 he was reluctant to change, because "the new scores might go down," 
and this might bring the wrath of the^ school board and community upon him 
. and the district (Interview, 2/5/80, p. 2)-. Another director of Research ^ 
.was so concerned that the test scares in his district might drop if he * 
changed .test!fe that he piloted the new test before presenting his -selection 
to the school board (Interview, 2/6/80, p. 3). Thus'sohoQl districts de- 
cide to select a new test only if there are strong reasons for doing so; 

they are more often inclined to keep using the standardized norm- 

\ * » 

referenced test they had previously selected. 
Who Is Involved in the Selection Process 

Once the decision has been made to select a* new test, most school 
' districts set up a- committee of representatives of test users. These m 
committees usually ^fompr-ise teachers, principals, central administration, 
and counselors. Generally a single person is in charge and guides the 



selection process, because he«oi; She is knowledgeable about testing — 
or. a€* least is considered sq. In large and medium-sized^^tropoldtan • 
school districts ,\ this' person is commonly 'the director or a A raember of 
an evaluation and Research unit, tn smaller and rural i&tricts that 
have no evaluation and testing staff, the person choseaMpight hold a r 
position in the' area of Title I pupil services,* cbunseling, curriculum* . 
development, or special education, . * . ' 

- Whatever the formal "role" of the gprson responsible . for selecting a, 

•A . • * ' ■ 

test, he or she enjoys a wide degree'of influence and discretionary power. 

■ *- * * 

For example, a principal who sa^ on the Selection committee.'of his la^ge 

city school district explained 1 that he had relatively Ifttle control ovef 

selection because "there is^nly pne' person trained 'in fche^ f ield here, * # 

and it's hard for.anyohe to compete tha't knowledge/ 1 He added -that 

the comndttee had been set up' on^Ly because "the administration is -committed 

to participatory man'Sgemen^, but it is not a democratic system when it comes 

to testing '. . . tnere is only one person who makes the decision and that s 

-our local expert" (Interview N 2/9/80, p. 3). * Those in charge of the 

selection process,, acknowledged their "local .expert" status. As one such 

person commented, "Although I consulted with others informally*,- \ pretty f 

much 'made' the decision" (Interview with General Supervisor, 3/13/80, ,p .5) . 

'But Although there was oile recognized or designated leader in the ^election 

process, those 'in charge usually convassedy the opinions and concerns of 

others in the district, either through the -committee or by less formal 

tpeans. As the head of an evaluation unit^in a city school district §x- f 

plained, "/the districtwide committee/ will say I made the de^isiorf, and 



T did, but I was also sensitive to them even if they did not know it" 

> 

(.Interview* 3/18/80, p. 2). - • • ' . 

Those guiding the 'selection process tended to insulate the .selection 
decision from the influence of private testing firms. ' For example,. one 
director .of research Who was in charge of test selection explained: "Al- 
though I talked with sales people from , private testing firms quite frequently 
before'we decided to buy a test . . • my- 'decision had much* more to do with 
ray own review of the material and Comparison's to other tests on the tnarket , 
*(Interyiew, 2/6-/80) , p. 6). In other c3se£, the person in charge was care- » 
ful to -limit contact between testing firm representatives and those involved 
in selection, .father than having representatives make presentations, for 
example, firms were*usually asked "to send.. . . their, materials for review 
and to answer . . . questions" (Interview with Director of Pupil Services, m 
12/12/75, P,.'l3). This was due to a concern, particularly prevalent in 
smaller and more rural school districts, that' testing firm representatives 
might sell them other than what they wanted. Private testing firms, on the 

r « 1 

0 

other hand, were quite willing — some more than others—to send representa- 
tives to make formal presentations to local school districts. 

The possible undue. influence of private testing firms is generally 
of less concern to those in more metropolitan school districts, often 
because they have the^.r own personnel knowledgeable investing. Thus-^it 
is more convenient for these districts to invite testing firm representa- 
tives to make formal presentations. But even here, .the invitation to make 
a sales presentation is usually extended only after the, test has been 



reviewed* by the person in charge,, and less thoroughly by others involved 
in the selection* And for the most part those who* select a test for dis- 
trict use do not regard private testing firm presentations as strongly ^ 
influencing" -the decision. Rather, these firms' representatives were seen 
as simply answering the questions and responding to the concerns of those 
involved. As a member of one selection committee 'explained, the presenta- 
tions just "provided us with information and pointed out the good features 
of the tests that helped us make a good decision/ 1 . She added that as th$ 
presentations were all "outstanding, 11 it was hard to say that they "had 
much influence" on the decision (Interview with^ Assistant Superintendent, 
5/23/80, p,*3).' One several-time participant in the local test selection 
process sunnjiari^ed this familiar process well*when he said, "While most 
school districts are, ostensibly at the' tnercy of private 'testing firms, 
they do their best to find out what they want to know before hand; 11 Be- 
cause of this, he continued, the role of most private testing firm repre- 
sentatives is usually "reactive" rather than one of hard sell (Interview 

b 

with Director of Research and Evaluation, 2/6/80, p. ty.* 



f 

* The allocation of -resources in the leading private testing firms suggests 
that they agree; tjhe sales budget is considerably lower than that spent 
on test development. Even in the firm with tfie largest sales staff, 
the- ratio between -sales cost and development cost is 1:4/ Thus these 
firms apparently attribute their ability to sell tests more tp the 
reputation of the .product than to salesmanship. There is wide variation, 
however, in th^ restive importance the major firms attribute to sales- 
manship. One company, for example, has a national staff of approximately 
60, while another maintains a staff of five. 



The consensus among those involved, in tiistrict test selection, then, 
is. that, they, make their own decisions and buy what -they need, rather than 
wh^t thef testing firms want to s^ll them. Private testing firm representa- 
tives have littief effect on those decisions because they are given little, 
if any, part in the selection -process. More often than not,^they are asked 
to make; presentations to ^he school board or a group, of teachers only after 
their test has been selected. Several school district personnel who take 
part in local selection have commented that the influence. of private testing 
firm representatives pales in comparison with that of j textbook salesman. „ 
A general supervisor .'in- charge , of both curriculum and testing in one dis- 
tfrict explained, for example, that—in contrast to selecting a test — she 
bad seen "a good textbook go by the wayside because the sales representa- 
tive macie a bad presentation, ^r a text . . ^ selected because somebody 
tfas very sophisticated and smooth* in his presentation 11 (Interview, 3/13/80, 

p. 6) . ■ ^ * 

If there is little evidence so far of circularity in\he marketplace, 
it may nevertheless be occurring in subtler ways than through direct in- 
fluence. Galbraith (1971),, Fromm (1955), and others have argued that 
most private selling in this .country is the result of sellers 1 shaping^ 
*of consumer*desires through mass communication, <e.g. , through advertising). 
And ptivate testing firms do advertise, often through professional 
journals and flyers sent by mail. It would be difficult to t show that 
these influences affect local decision making; but neither can it be 
Shown that they are ineffective, or that circularity is clearly not at 
wark in school districts 1 decisions to buy tests— though school officials 
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are probably not consciously aware of it.* Thus our tentative conclusion 
is what the available evidence suggests: little circularity is occurring ' 
in the educational testing marketplace, and school districts largely make 
their own decisions atiout the selection of a test. ^ 

But this still ,tells us little about the bSsis £or their selection 
or their rationality as consumers. This remains an important quelle ion 
because, to' the extent that they are rational consumers basing their' 
choice on th£ir needs and preferences, school districts can be seen as 
influential and powerful in the marketplace. The next section of the 
essay therefore 'focuses on the criteria by which school districts select 

Vta.tr. \ , Ji 

.THE BASIS OF 'SE^cAoN ' 

" ft 

The basis upoh which a school district "selects a standardized norm- 



referenced achievement test depends in part on that district's needs 

/ . 

and circumstances. However, since district officials are commonly inter 



r- 



ested in 'selecting a test that will enable their school boards and com^ 

munities to cpmpare their district's student achievement with that of 

others ' across the nation, they initially select a number of candidate 
• * 

tests ^hat seem likely to serve that purpose — that is,* on the basis *of t 
technical quality. For example, the director of research in one school 
district explained that what went, into his choice ot candidate tests was 



their "technical quality/ 1 Similarly,, an assistant superintendent of 
instruction said that th^ district's former director of research had 



* This suggests that if circularity is occurring, it is impossible to 
determine what weight, if 'any, it should be given in this analysis. 
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shortened the committee 1 s list of possible tests Vsimply on the basis of 

technical considerations 11 (Interview with Administrative Assistant to 

the Superintendent of instruction, 5/23/80, p. 3). 
* * 
• The number of tests that are initially considered ranges from two 

to eight. Typically, the same four or five tests , ^provided by the leading 

major testing firms, are regarded as-the technically better tests, and 

are therefore more seriously considered.* These* four ,or five are usually 

considered to be of better technical quality in two ways, not mutually 

r . 

exclusive-— their reputation, aod certain methods employed for th^ir con- 
struction. 

Many school officials, .especially in smaller and rural school dis- 
tricts, simply rely on the reputation* of tests and their use by neighboring 
and metropolitan districts. For example, a director of curriculum in 



'charge of the local test selection effort was asked how he knew that the 

V 

three tests initially considered for adoption were technically reliable. 
He explained that they had been ".statistically checked" by others. He' 
added that, for him, selecting a test — at least initially— was like 
"buying a car— -youxask other people whose qpinion you respect and then 
you 'form an opinion of what is a .good car" (Interview, 12/13/79, p. 4). 
In this case, he relied on some local experts in surrounding school dis- 
tricts to form his opinion of a technically sound test._ / 



* These tests include the California Test of Basic Skills, the California * 
Achievement Test, the Iowa Test of Basic Skills, the Metropolitan 
"Achievement Test, and the Stanford Achievement Test, and are published • 
by four testing firms: CTB/McGraw-Hill; -The Riversitfe Publishing 
Company, a subsidiary of Houghton Mifflin; the Psychological Corpora- 
tion, 4 a subsidiary Qf -Harcourt Brace Jovanovich; and Science Research' 
Associates, a subsidiary of IBM. 
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Those in charge of the selection process in more metropolitan school 

districts typically relied, less on reputation and presumed quality. Asked 

how, they selected candidate . test's, they pointed mainly to the norming 

\ 

characteristics of the leading handful of tests. These included how a 
test had been nationally normed — that is, whether there was evidence tfhat 
the norming sample of students has been carefully chosen to represent the 
composition of the national population according to such variables as 
location, race, sex, and socioeconomic level; and how the content of the 
test was determined — that is, whether there was evidence that the test 
content was' dra^n from an extensive review of curricula used across the 
country so as to simulate a national curriculum as accurately as possible, 
A director of research conveyed this when he said that he checked whether 
tests he considered had "norming practices that were at least sound and 
/wereT going to provide a reasonable curriculum base* of comparison 11 
(Interview, 2/6/80, p. A) % . Sometimes other construction features %re 
. taken into account. For example, an assistant superintendent in charge 
of test selection made sure that a test had been not only properly normed, 
but adequately "field tested," so that he knew it would work (interview, 
12/13/79, pp. 8, 9). But the preponderance of attention is usually given 
to their norming practices. 

m 

Typically , 1 the norming practices of the leading four'or five tests 
are checked without reference to published professional reviews of the 
tests. A director of curriculum who was in charge of local test selection, 
when asked whether his committee paid attention to such reviews, responded,' 
"The hell with consumer reports." He added that in his district it was a , 
matter- of finding out what other people thought was good .(Interview, 
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12V13/79, p. 4). This rather common attitude' aside, outside* pro-* 



fessional reviews generally regard the same tests "a5 indeed the better 
Qnstructed .($uros, 1972; Center»for tfie Study of Evaluation, 1975). 

If .this consensus may make the choice of candidates easy, Jiowever, 
it makes the choice among them difficult. Most school officials see the . 
ieading tests' as technically similar and are unable to make any significant 
distinction among them. As one program evaluator put it, because "all 

the tests were pretty much the same as Jar as standardization was ccm- 

' • > ' • \ 

cerned, 4 ' "t-fiere "Was little to recommend one over another (Interview, \ 

.2/4/80, p. 2). 'Another local official explained t^hat "all these instru- 
ments are about tl?e same; they are all widely normed, and trying to make * 
airy determination of differences woulcf- have been a pain in the ass 11 • 
(Interview with Director of Pupil Services, 12/12/79, p. 2.). 

Because t!l!ese tests are technically rather similar, local school 
officials' problems with them are often the same. One common complaint 
is that the test content of 'the leading tests does not match the local 
curriculum; and that the tests -therefore do not measure what is taught* 
Ofte reading • consultant , for example, found that because of the lack of 
fit between the local curriculum'' and that used by standardized tests, 
,f if wy/try to plade a child according to . . . achievement tefst scores 
with the materials /used/ for that grade level, the- child would have 
.lots of problems" (Interview, 5/28/80, p, 2). Similarly*, an English 
teacher explained that the teachers .in his school are "basically 

opposed to testing because it measures what' they don^t think they are 

« *. ' • * 

teaching" (Interview, 1/31/80, p.; 3). ; ; 

' Another frequent • complaint is that these tests produce inconsistent 

» I 

results across grade levels. For example, a program evaluator hadj k 

\ ' . '21 



18 



serious reservations about usin£ only one test because student^ seemed k 
to score substantially low^r^ith it, afhigtier grades .(Interview, 1/31/80, 
p. 2). Pointing to the same ]>roblem > : ^<WKdirector of testing in another 
district explained that using, a newei* version of a test, students in -the 
third, seventh, an? eighth graded scored on average two to three grade' 
equivalents higher than with, the old version. Meanwhile, students itf 
the other grades scored near grade levelji as they Jha4 with the old 
version (Interview, */23/80, p. *2) . * * • 

. Often, these tests are. alsd- regarded" *as not relevant to the general 
level pf achievement in the district*, Students in a 'given school dis- 
tirict, for example, might uniformly score off the bottop or.-tthe, top of 
the test. A local school -of ficial in a district where mpst students 
typically scored at the bottom of the test called the test simply |n 
"experience in frustration" for the students, while teelirig teachers ; 
little about individual student achievement le.vels (Interview with 
Research Department member, 2/1/80, p. 5). * In* another school di%tridt,'- 



1, 



here most students typically sco^ itt the ninetieth percentile, the 

director qI testing complained of the same" problem— namely , that the 

v. - » 

test gives little indication of individual differences in student 
achievement (Interview with Director* of '.Research and Testing, 2/5/80, 

p. 2) . <& * 

The technical basis for these complaints is, for the most part, 
documented in the professional literature Ion 'testing. For example, it 
is often pointed out that standardized norm-ref.erenced tests inadequately 
reflect curricular , content and the objectives, of schools and school dis- 



tricts around the country (Madaus, 1980; see also Porter, 1980'; 
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Averch et al., 1972; Stake, 1972). Furthermore, critics charge that 
the.re is usually little correlation between standardized achievemeht 

* test resul|^ and measures more directly related to instruction (Stadolsky, 

~* 

1972; Madaus & Rippey, 1966). 

The literature also- points out that &£cause standardized achieve- 

» 

ment tests are constructed to differentiate maximally among individual' ' 
' students, their accuracy in testing group performance and performance 

* across groups- is reduced (Carver, 1975; Porter & McDaniels, 1974). .This 

- > / 

might ^account for dissatisfaction with inconsistent comparisons o f »fijgU- 

« * > 

dents in different grades; It is also argued that because standardized 
vv tests need to establish a normal test distribution ,^they cannot reliably 
measure achievement at the extremes *of the distribution (Tyler,* 1974)-- 
which helps .explain why officials in schools with mostly high or low 
achievers often complain that the tests provide lft tie information on 
Achievement differences'. 

Since the^ leading and better-constructed tests share some of these 
short comings, local school districts cannot easily choose among them on 
the basis of their norming practices. The most commoti problem that , 
local" school officials do try to address is that of the fit between a 
test and~their curriculum. 'But this is difficult to do in a sound 
and systematic way — at least with a limited amount of time and money. 
As one local test expert explained^ you simply "could not pick a test 
based on the curriculum ... it has to be done on a trial and error 
j b'asis \ . . there is just not enough time £nd energy for> someone to 
. go through and do this in any sort of comprehensive- way" (Interview v 
with Director of ^Pupil Services and Special Education, 5/23/80, p. '5). 
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Not atypically, when comparisons were attempted they often turned up, 
"very iit;tl<r~ctif ference" among tests (Interview with 'Assistant Super- 
intenden.t, -5/23/80, p. 2). And professional reviews of these> tests, even ,< 
if they &ere conducted, * would not o£fer much help, since they ^re concerned 
with the broader aspects of test constructidfi rather tshan the particular!- 
ties of school district curricula. 

School districts, then, are rarely entirely pleased -wit h the- leading 
^id technically superior tests; nor do they .find it easy to select among 
them on the basis of ttieir .-norming practice. One might thus argue that 
school districts are limited in their choice and their influence in the 
marketplace because they cannot buy exactly what they want. Yet school 4 
districts continue to btfy these tests because district .officials belief - 
that * they are basically sound and provide the best available nationally 
comparative information on student achievement. On th^ basis, one might 
argue that school districts do have some choice in the Marketplace and 
have been able to influence private testing* firms through those choices.' 
If, that* were not so, one would expect little attention to pe paid. to 
the technical reputation of tests or to the use of certain \norming 

practices%n their construction; one would expect any test Claiming to 

\ \ 

• l - 

be standardized and nora-ref erenced feo be considered on a pair with the 

* \ • 

leading four 'or five. ' \ 

' \ 

Although the norming practices employed by the leading tfcsts are 

similar, this does not -mean that their tests are all the same3 There 

m 

is evidence that the tes^ts .differ in their ability "to meet other school- 
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district needs; and it is on this Jtesis , unrelated- to norming methods, 

* • 

that major testing firms compete with one another. An experienced hand 
in local test selection efforts summed this up well when he said, 
"Since there is such little difference in the norming practices with the 

• - - ' • - v 

large martuf actuters of tests, it's the other stuff that becomes important 

(Interview with Director of Research and Evaluation, 2/6/80, p. 6). 'And 
<s * 

on this "other" basis, consumers see t^e major private testing *f inns as 
quite different an4 competitive. y t * ■ , ■ * . ' ' "> 

Selection Considerations^nrelated to v ^brming Methods 

'Among' the cons.idera^ons unrelated 'to norming methods are a test's 
ability -to cover a wide range of student achievement and its ability 
to provide information that is seen as helpful to instruction. Other* 
test attributes include the length of administration and the appearance 

a 

of th€ test. The supporting services provided by the test publisher, such 
as promptness in scoring and returning test ^results and the ability to t 
provide needed technical assistance^, may also be .considered. Several of 
the'se attributes may enter into a district f s selection decision; one or 
two of them may stand out' because of a particular "concern or bias^ of ^the 
•local, test expert. But local test selection decisions may well:-also 
represent, formally or informally, *a composite of various considerations 
within a singly school district. -J J 

Interestingly, one of the traditional criteria for consumer purchasing, 
*the ;£OSt of the produdt, rarely has much influence in the selection decision. 
In part, this can be explained by .the relatively uniform costs of the tests 
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put out by-fcH& major publishers, as well as by the relatively small pey- 
- centage of school district budgets that is spent on testing' programs'. » 
However, it is not uncommon for § 4-ocal school district*, having ^made a , 
decision to buy a certain test, to- negotiate for a lower price. 

What follows' is a discussion of some prominent considerations that • 

may^enter into the ±ocQ selection decision — certain ^attributes of a 

f •* ' f • 

test, and services provided^ by te$t .publishers* . *■ 

Test Attributes 

As pointed out earlier, the leading four or five tests share common 
^normingjyethods and this makes it difficult for school district consumers 
to select between them.' Those selecting tests for school districts 
though often select that test whi^h has been more recently normed. This 
is desirable since it promises to^rovi4&"compari,sons of student achieve- 
ment with the most current national levels of achievement. Typically, 
this is a concern of- the local tefet expert, usually an evaluator or member 
of the research department. Depending on his influence and the. prominence 
of other local staff concerns, this may ^be the major reason why d certain 
test is selected. In one school district, the curriculum coordinator 
* recalled that of the two tests they were considering, "one of thenj had 
• older norms, 11 and was therefore considered "no good" by the research de- 
paftment (Interview, 1/31/80, p. 2).. 'Similarly, a math consultant- in the 
department of curriculum reported that the committee selected thfxfest 
they did mainly "because it represented the newest norm version of tests 
available" (Interview,- 2/1/80, p. 1). If one of the leading tests was 

I 
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recently renormecl, it might thus* be more favorably considered in a 

district seeking a new test. This was illustrated by a member of a 

research department in a metropolitan/ school district who explained* wjiy' 

his test selection committee, chose a particular test: "fcjie other jnajor 

manufacturers were just getting ready to put their new tests on jthe • 

market, but this one had already been renormed and was in operation — 

so we, went with it. 11 (Interview, 2/1/80, p.2 ). Because private testing 

firms recognize this,- they compete to put^newly normed and updated tests 

on the market. This cannot be done too frequently, however/ because the 

development and construction of a standardized norm-referenced test is* 

estimated take approximately four years (Interview <with Director of 

Research at CTB/McGraw-Hill, 2/7/80, p. 3). - - * , % 

-When during the school year a test was empirically .normed may also 

be' an important' consideration to the ideal test expert. This is because 

the, selection of a £est normed 'at the time or times of the year when 

local testing occurs promises to provide more accurate cpmparisons with 

% national norms. As .one local test expert, a director of special services, 

recalled of his tenure, on the test selection committee, a test was ehosen 

tfcat: had "two . . norming dates, and this compared .faVorablj? with most ^ 

other standardized tests,- which only had one norming time." -This was. t 

seen as desirable since 'it gave the school district greatef flexibility ' 

In deciding when to test, without having\to sacrifice any -'accuracy in 

*• •* 

test results, by testing- at a t^me different from that at which the 

national norms were* determined (Interview, 2/5/80, p..l). Typically, 

this consideration was important in larger school districts with evaluation 



and testing specialists who^ere interested in mjlximizing the accuracy of 

district test result^ In school districts without a test specialist u the 

norming of tests, was accorded little importance and rarely entered l^to 

selection decisions. ' • , 

Another attractive test attribute- is the ability of a single subtest 

of a K-12 battery of achievement tests to measure across* two or three ^ 

grade levels. • Tliis allows students greater opportunity to be tesped at 

their level of achievement rather than simply -at their grade placement , # 

» 

and is especially valuable where ^ student f s achievement level, is several 
grades higher or lower than those of his or her cohort in the same grade. 
For example, a program evaluator explained that "one reason for selecting 
one of two leading tests was that it had "multi-level testing." This was 
a particularly important consideration for his school district because the 
students generally scored below grade level and "they got crazy results 
when they had to test ehem only on . one grade level. 1 ' By crazy he meant 
that most students scored 'uniformly at the bottom* (Interview, 1/31/80, p.l) 

^ The multi-level ^apability of a test were particularly attractive 
to, dkininistrators and teachers. Administrators liked it because it simpli- 
fie % d the test ^ninistration; one test could bje given for several grade 
levelS; Teachers often liked it because it allowerd their less advanced ■ 
students to answer questions at a lower grade level, and therefore not to 
feel overwhelmed and frustrated by the test. 

' One of the leading tests may also be selected because .of other 
attributes, s'uptv as ability to aid in 'classroom instruction. A test 
may, for example? allow the scoring of groups* of • students in terms of 
mastery of certain skills. Thus the .director of test selection in one 
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district sought a test that "teachers would use"; otherwise, there was 
no purpose in testing.* With this in mind, he explained, one of the reasons 
he ^elected the test he did wa^ teachers f response to a survey he conducted. 
The teachers liked a particular test because it had "a profile breakdown — 

that is, it related certain items to the instructional objectives identi- 

• • « v 
fied." According to him, the profile breakdown would enable teachers to 

•« , . « i 

see- how a class performed on the test according to various sl^ill objectives 

* * 
and so help them identify what areas to* stress in their, teaching (Interview 

with Director of Research ancT Evaluation, 2/6/30, p. 2). A related feature 

9 

that some considered important in promoting instructional use was-a test f s 
ability to provide specific diagnostic and prescriptive information. For 
example, an -instructional specialist iin reading explained that she and 
the teachers on the selection committee wanted a test offering the type 
o£ feedback provided by the mini-tests at the back of some reading series— r 
on areas in which the students we/e weak, and on what part of the test they 
'could practice to improve. The committee therefore selected the standardized 
norm-referenced test that provided the most detailed data on individual 
student performance, in the hope that it would be useful fpr "remedial" 
instruction (Interview, 3/13/80, p. 2) + The test .selected in this case 
provided references to several major textbooks that corresponded to the 
areas of* weakness/ indicated by the test results* And in some districts, 
administrators — both principals and central office staff— hoped that -rising 
a test capable of improving instruction would result in students 1 scores 
going up — and that the school 'staff then would look good in the eyes of 
the community. Teachers, however, were typically more interested in 
• t^l -usefulness of test results for classroom instruction. * 
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Administrators and teachers were also generally attracted to 
another tfest "attribute : the provision of student achievement scores in „ 
comparison to an "ability" or "anticipated" score (the latter being b^sed 
on an intelligence test contained within the achievement test). This 
score is usually seen as potentially useful in teachers 1 instructional 
endeavors. One principal called i^ a "helpful tool" to teachers, which, 
could *serye as a "validity check," to determine whether a -student was 
doing what he or she was capable pf (Interview, 5/14/80, p. 2). Simi- 
larly, a teacher explained that he an'd his colleagues on the testing 
committee were attracted to the "anticipated score" because it enabled 
teachers to "really get down and^help the student" (Interview, 12/18/79, 
p. 7). Until recently this feature was offered by only one of th§ major 
private testing firms; and because it can be an aid to instruction, it 
was influential in a large number of school district selection decisions. 

Other considerations that may enter into local selection include the 
time required to administer a test, or even its appearance. Of these, 

test length was the mdst frequently mentioned. For example, as a member 

f 

€ 

"of one selection committee explained, most of the teachers wanted a test 
similar to but shorted than the 'one they we're using (Interview with • 

'Director of Special Services, 2/5/80, p. 1). Similarly, one reason why m 
a principal pushed for the test eventually adopted wa$ that it could^ 
"get in and get out ai\d did not take long for kids to db" (Interview 
with Principal, 5/14/80, p. 1). Moreover, particularly in the lower 
grades, teachers "did*tiot want to have tt^ir kids sit for so many hours 
taking -/the test/" (Interview with Research Coordinator, 2/1/80, p. 2). 
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As these examples indicate, the general feeling was that the shorter the 



test, the bettejr. Typically, this was se|g as important by teachers and 
• principals who joften felt that testing took thea^ away from providing more , 

I ' 
direct instructional services. 

The appearance of a test was sometimes mentioned by teachers as a 

cbnsideration; in seleption. An assi's N t^nt superintendent o*f instruction 

, said that one u of the subjects discussed iri^ their committee,' composed of 

teachers, was the format of the (est and whether it was "in red, blue, or 

green." She added that this sounded rather simplistic, but "was important 

because it was difficult, for example, to read certain tests which we*e 

printed in red . . . on white paper 11 (Interview, 5/23/80, pp. 22-23)\ 

A member of another test selection committee explained that one of the 

•••hot issue3 H that emerged was the size of the print. He commented that 

it was "funny" that this should 'Vend/ up playing a role "in the," decision," 

that that it was apparently one of the reasons teachers did not like the 

old test (Interview with Administrative 'Assistant,' 5/23/80, p. 5). In 

smaller and more rural school districts, often without professional testing 

and evaluation staff, such test attributes as a short time of administration 

and an easy-to-read" format played. an important role in the selection 

decision — usually made by principals and teachers. 

, Service by Test Publishers 

J 

Another consideration in. the Selection decision, generally less • 
^important' than those discussed above, is the kind and quality of services 
that the major testing, firms provide with their tests. One such service, 
^usually ol more interest to administrators and evaluators than to teachers, 



is test manufacturers 1 prompt return of test. scores to school districts. 
For example, a director of curriculum, who had sat on several test selection 
committees, called the "quick turn-around time for a test" a common concern 
(Interview, 2/4/80, p. 2), because it allowed the placement of students at 
the beginning of tne school year, ox the checking of their achievement at 
its end. Thus, when. a new test was 'selected the testing firm that co\il$ 
demonstrate a potentially 'short turn-around time usually heW a competi- 
tive advantage. In districts that scored their own test or made scoring 
arrangements independently of the test publisher, of course, this considera- 



\ 



tion* did not enter into selection decisions. 

% District personnel also commonly mentioned the availability of technical 

assistance or "consultant seiVides" from private testing firms as a considera- 

tion. For example, the director of testing in a large city school district 

wanted to find a testing firm whose representative he could call "and have.* 

* t> 
him come down to help-sjne put in a new order' for next year or help me sort 

otrt some problems , with the testing programs." He added th'ajs he was simply 
looking for a testing firm that followed good business "practice — "not 'to 
sell and run" (Interview, 12/18/79, pp. 8-9).* In some other school dis- 
tricts, the availability, of technical assistance for such matters as test 
interpretation, workshops for teachers, and presentations to the board also 
could' in f>uerice test selection, though commonly it was secondary to other 
c ons i de rat ibns . 

The range of considerations^^ which school districts ultimately 
select among .the leading four or five/ tests is trhus quite wide, and varies 

v" 

greatly across and Within school districts.* ^ven within a single school 

district, different people involved in selection* decisions may agree on \ x 

« i 

• » " • 
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a given test for different reasons. For example, in one school district 

central administrators and Title I personnel were interested in a test 

primarily because its. publisher promised to. Score and return it promptly; 

and principals and teachers tended tpward tt\* same test becaus^it was 

short and easy to administer. In this example, ^he test was selected 

because it combined features that were attraptive to* the different staff 

inyolved in selection* In other cases, those involved in a selection 

decision shared a primary interest in certain test attributes and selected 

* f 
a test on that basis. For example, a school district with^a strong commit 

ment to using test results for improving instruct!^ was particularly 

attracted to. that norm-referenced test which they beVi^eved provided the 

best breakdown, of student achievement according to specific skills. 

Through this process of sorting and weighing th'd various features- 



and capabilities th"e leading tests provide, districts 
test that comes closest to meeting their usually fflgtented needs and 



thus settle on the 



wants. Examination of that process made it clear. tha: the leading private 
testing firms provide different options and services :o meet those needs 
and wants. Typically, those that offer the more attractive .options and 
services enjoy the business of more school districts. Indeed, a review 
of the past tests of -these firms shows that they have been Changed in 

numerous ways so as to make them more appealing and niore responsive to 

j 

consumers 1 needs. Efforts have been made to ensure that the tests are' 
relevant to and useful for instruction. ' One private 
'particular has added several new features to its ^ tests, and has been 
successful in capturing the lion f s share of the market. 

However well tliey may meet the iftternal requirements of school di£- 
trists, the purchasing of- tests from private testing firms fs also 



testing firm in 
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influenced by state and federal requirements. The nature and extend of 
that influence is discussed below. 

State and Federal TestiW Requirements 

Clearly, the stronger influence on local test selection is 'that of . 

* 

state's testing requirements that mandate the use of a particular norm- 
referenced. test for certain grades.* This influence is strong because 
local school districts, with few exceptions, use the same test for local 
testing s in other grades. 'As an assistant superintendent explained, his 
district decided to select a particular test for grades K-12 "basically" 

9 

because "the <state had selected a test'^and mandated its use in grades 4, 
8, and 11 (Interview, 3/14/80, p. 1), Similarly, a district; director of 
testing .said that the state mandate of a norm-referenced achievement test 

. was the determining factor in their decision: since "we were using it to 
meet the state mandate already, it seemed the natural thing to stick 
with it" (Interview, 2/4/80, p. 1).- 

A state-mandated test is influential for several reasons • One is 
that the test results are often used by the public, if not the state, as 

' a way of evaluating local school districts. Thus, it makes sense for a 
school district to use the same test as the state, so 'that any curri^ular 
or instructional weaknesses can be identified and addressed, A second 



* States may also be influential in local selection decisions by providing 
a list of -approved tesfcs, as is the case in New York and Wisconsin. 
Typically, this list includes the same tests that are ordinarily con- 
sidered to be of superior technical quality bjr lc>cal school districts* 
Tliough no districts in states with such lists #ere' visited in this 
% study, 'it is likely that their selection is based also on the considera- 
tions discussed in this essay. 
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reason is tftat local school officials, especially f in smaller districts; 
are open to the influence of the sPtate since it diminishes their responsi- 
bility for selecting a test/ The major private testing firms are aware of 
thi^, and consequently the^e is much competition among them for state 
selection of their tests/ Winning a state nomination greatly increases t 
the likelihood that school districts in that state'will select the same 
test for their testing programs. 

In mandating a certain test, however, the state may be ^responding in 

part to local preference. For example*, an associate director of testing 

v ■ h * * ' 

who* was in charge of the state f s test selection said that t'hey had chosen 

•* . » 

a particular test because "this test and testing firm was, familiar to local 
school districts and had already done a very good job" (Interview, 3/10/80, 
p. 2).. State officials also are inflijenced by local officials who^ sit on 
the state selection committee, and* who more often than not are drawn from 
a large city or cities -and are knowledgeable and politically powerful. 

si • \ 

Thus school d£strict;s have considerable influence on test choi'ce &t the 
state level. 

Federal evaluation requirements / especially for Title I,' can also 
enter into school districts 1 test selection. But typically, because no 
one test is mandated by the federal government, that influence is indirect. 
School districts may. decide to select for local testing programs the same _ 
test Jjhat is used for evaluating federal programs such as Title I. This 
enables them to use a single test for their,major testing needs : rather 
than having to test students enrolled in Title I with a different test 
than that used for other students. . For example, in one such school system 
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a member of the test selection- committee,, asked #bout their basis for 

selection, replied: "The Title^ I people seamed to- have some rather *|irm 

* * • - . % 

ideas about what they needed," and ^added, "They had certain constraints 

we had to operate under" (Interview with Codrdinato^r of Curriculum K-12, 

3/31/80, p. 1). ' The constraints Imposed by testing for Title I. programs 

often include an effort to match the norming dates of tests and those of 

school districts. For example, an assistant director x>f evaluation in a 

t 

large school district pushed' for one test, which was eventually adopted, 
because "it had dual norming (fetes and -was easier to fit into /Title ll 
guidelines which required testing within thr 4 ee wefeks of the> norming date" 
(Interview, 2/14/80, p. 1). Similarly, a member of attest selection com- 
mittee in a large city school district explained thaf they had previously. 



selected a tpst partly because*, it "was^.normed^in the fall and spring and' 
therefore could be used for Title I pre- and po%t-testing" (Interview 
with Research Coordinator, 2/1780, p. 2). .'And an evaluator for special 
projects, including Title I, uanted to make sure that the test selected 
by his committee had a "diagnostic component," so that the program people 
could use the results for providing -remedial. ins tructio'n (Interview with 
Coordinator of Special Projects, 2/6/80, p. 2).^ 

In those cases, then, where' school district "testing was merged with 
testing for Title I, the particular needs of Title I testing played a part 
in the selection of a test fou local programs. Private testing firms, 
aware that a test meeting Title I requirements may also -be used for local 
Resting programs, thus compete quite- intensively for the. Title I market. 



While federal — especially Title I — and state testing 'requirements 
can be influential i^local test selection, school districts control the 
extent of that influence. Even state selection of mandated norm- 
referenced tests is based partly on previous local selections and on poli- 
tical factors. Furthermore, school district^ ne^d not u£e the state 
^andated test for their own testing if they consider, it inappropriate. 
Similarly, the influence of Title I testing requirements is limited to 
local districts 1 selection of tests for Title I evaluation, unless dis- 
tricts decide to use the same test for their own testing. And even then, 
district personnel can select that test which best me^ts some combination 
of their Title I-and district testing needs. Thus school districts, while 
influenced directly and indirectly by sjtate and federal requirements, are 
still able " to exercise their preference as consumers in the marketplace. 

THE INFLUENCE OF- SCHOOL. DISTRICTS AS CONSUMERS ' 

This account of how and on what basis local .school districts go 
about selecting educational achievement tests suggests several observa7 
tions about their influence as consumers. On the one hand, school dis- 
tricts can and do exercise choice in the marketplace, and so are able to 
inflneaee* the tests produced by private testing firms. TheiV power as 
consumers is shown by the fact that these firms provide technically 
well-constructed tests — at 'least as far as certain norming practices 
and procedures are concerned — that enable them to compare the achievement 
of their own students with that of ttye- larger national population. And 
even though this selection criterion has seemingly limited the number of 
testing £ inns to a handful that share a virtual monopoly over the norming 

37 



of such tests, there is much evidence that' districts select among the 
leading tests on such grounds as their abiflity to imprpve classroom in- 
struction, to simplify the administration of testing/ aiid to meet state 
or federal te'sting requirements, to mention a few. That such choices 

- , ■ * i 

are oJ¥ered indicates that school district needs influence the fcinds of 
tests}, produced. Otherwise, one would expect tests to differ' vefry little 

* . • J . 

from one another, and districts to take any test at random, having no 
basis for selecting one over another. Similarly, one would not expect 
private testing firms to compete with one another at all. 

On the other hand, this account suggests that while school districts 
enjoy a good deal of choice and i^fjLuence in determining some aspects of 
the tests they buy, they are limited in their Influence over and satis- 

' % 

♦faction with the way tests are normed. They cannot know how accurate 

I 

the information is that tests provide on comparative student and local 
school achievement, or whether it does, in fact, meet their needs. For 
example, they- cannot systematically determine whether and to what extent 
the tests developed by the major private testing firms take into account 
what they are teaching their students. in their schools and classrooms. , 
Indeed^ there is some evidence that' most local school practitioners, 
especially teachers K think that these tests bear little relation to what 
is taught in their <^lassrooms; at best, they see the tests as broadly 
and vaguely related (Kennedy, Apling, & Neuman, 1980). Similarly, school 
districts are unable to know or determine to what extent riorming samples 
of "these tes-ts are, representative of students in their district. The 
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student population of most districts rarely matches^ that of a stratified - 
"random sample of the national student population used by grivate testing 
firms'. Nevertheless, the achievement of their students is compared t to 
that of the sample population. With .respect to test construction, then, 
school districts 1 influence over private testing firms is limited. ^ 
This discussion thus far has dealt -with the choices school districts 

i; « 

have and. their ability to influence private testing firjns by exercising 
. those choices. Arid it has pointed out that, to the extent that they can- 
not exercise choice, their ability to have testing f^rms produce what 
they want, and toxoid them accountable, is limited. 

It has been assumed "throughout that school districts are rational 
consumers who know what they want and express their^jpreferences through 
choices in the marketplace. But perhaps school districts are less than 
rational consumers. For^Va^ious reasons, districts may be unclear or 
inconsistent about what" they want. They want, for^e'xample, to compare 
local .students 1 educational achievement to that of students nationwide — 
and yet want a test that will reflect the local curriculum* and student 
population. And the two are inherently incompatible, since there is 
no national curriculum or standard student population upon which a test 
can be based. 

School districts, then, may be shopping for what does not exist, 
tt i's not £hat they are simply asking for too much — apples and oranges 
* to be put in the same basket — but that they want what cannot -be provided — 
fire and ice in the Same container. It may be in the nature of, the local 

r « • 

education enterprise that they are bound to want both — to see what v their 



MC 




3 



own students are learning from what is taught 4 and to compare this 

objectively to what other students are learning and are teing -taught. 

But since school districts have different curricula and are composed of 
i 

differe'nt student populations, school distric ts can not have both. If . 
this is indeed the case, school districts could not get what they want 
from these tests 'even , if * they exercised more control in the marketplace. 

THE MARKETPLACE AS^A MEANS OF LOCAL CONTROL * 
Given the nature of the local educational enteifpiise and its not 
wholly rational wants, 'it is unclear whether the marketplace can work 
better. School districts want a test that can standardize what are 
essentially local -conditions so that an objective. comparison of student 
achievement can be provided. ^Private testing firms have responded to 
this demand; given its irrational nature, it is difficult to see in 
whatL ways they could be still more responsive. For example > as said 
earlier, no educational achievement test can be constructed that 'matches 
both the curriculum of any single local school "district and the Curricula 
used in all other school districts around the coyntry. Private testing 
f irms^already make an extended effort to take account ^ of the various 
curricula, texts/ and instructional techniques of different school dis- 
tricts. In fact, evidence of this endeavor in Standardized norm- 
referenced; achievement tests is one criterion upon which school dis- 
tricts base their selection of a test in the first/place, furthermore, 
where school districts are clear about the intandid use of a test (e.g.; 
for improving classroom instruction, meeting stafte and federal require- 

St i j 

ments. or administrative accountability), they can select a test con- 

» 

taining those features that best meet tiheir needs. And school districts 
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who know th$y waat a .test that is relevant to and compatible with their 

curriculum can additionally buy criterion- and objective-referenced tests 
* 

from the same or other privatfe testing ♦firtis. K Indeed, the fact that * 
school districts have expressed a .need for -this type of test 

0 ' 

(R&fccziiiski, 1977; Ebel, 197^ has contributed tor a rapid increase 6f 
new i testing firms and tests on the market/ x 

The marketplace as a means of ^local control, jthen, may work insofar 
as "consumers know what 'they want and can thereby hold private testing 

4 * 

firms accountable. Where school districts find that their nteeds and 

wants are not met through the marketplace^this* may have less to dp 

with their power — or l^ck of it — as cdnsumers than with the rationality 

* * 

/of these," needs. Priva^p testing firm development and provision of 
'standardized norm- referenced achievement tests may simply ^reflect mixed 
or inconsistent mess^es from school* districts. 

/ * ' ; 

Private testirig % firms operating in the marketplace may thus well 

* • * ' - 

be the- best vehicle for maximizing local control. They play a unique, 

if imperfect, role in taking account of local educational conditions f 

and at; the same time providing a. neutral base for assessing national 

student achievement.. *It 'is not c^ear that tlier^/ are alternative 

governance arrangements that could better provide, this delicate balance. 

But v before assuming such a conclusion, it Is best to consider some possib; 

, arrangements. 



THE MARKETPLACE VS FORKAL^G£VERNANCE ARRANGEMENTS 

1 One alternative is fpr local school boards ox; their representatives 
to develop their own tests • This would have the advantage that the test 
would be based on their own .curriculum and stu^gnt population. But there 
•are" several problems with this. First, districts would likely be unabl.e 

to provide much of a comparative base for assessing their student achieve- 

> ** *» « 

ment vis-a-vis that of otjier School districts in the qbuntty.. In addition, 
they probably lack the technical! ability, manpower, and other resources 
%o provide a technically credible^Sial^dT assessment, and may thus require 
outside expertise. This may be particularly; the case with 'smaller school 
districts jthat.do not have the- staff .for in-house test- development efforts. 
Furthermore, test development by local school boards and their representa- 
tives may involve extended ^political debate and conflict with others in 

the district over both substantive and methodological issues. This has v 
v * ' 

already been a problem .with some large school district efforts to develop 
their own objective- and criterion-referenced tests- (Interview with 

* i 

Director of Research, 1/2/81, pp. 2-4; Interview with Program Evaluator 
in another school district, 1/1/81, p. 3). . ' 

Current market arrangements avoid^ many of these problems. Private 
testing firms, by virtue of their national role, have access to a wide 
variety of School districts and students for comparing achievement and 
,can^i^/elop tools of assessment^at relatively low cost because of the 
large- number of consumers. "They also 9ffei\£n independent -and authori- 
tative base for comparisons among' school districts and^ thereby minimize 
political -and public debate at the local level. ' 
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^Another alternative to the existing market arrangement is for 
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states^ to ^assume more responsibility for developing educational achieve- 
ment tests. An advantage of. this approach is that state education 

/ ' - ' 

agencies are possibly more in touch with and sensitive to local school 

district curricula and student populations^ than are national^organiza- 

tions like private testing firms. In addition, state agencies are 

v 

formally accountable: to elected officials, so that school districts 
can lobby for changes in the way tests are constructed. Furthermore, 
many state agencies have the necessary resources for developing such 

tests. t * - 

- A major disadvantage of this alternative is that state agencies 
would be unlikely' to, be able to develop a standard test that can provide 
much of a national, comparison. "More likely, each state would be inclined 
to develop its own state test, rather than national achievement^t^sts, 

a$ has already been" done, for example, in California and Texas (Ebel, 

■* • 

1974; Pipho, 1978). Additionally, these tests are likely to be geared 
primarily to meeting state needs — such as accountability — rather than 
those of local school districts. There is already evidence of this 

occurring in California and Texas school districts 1 reactions to the 

1 ■ - x ' 

state-developed .tests; they typically buy' standardized norm-re,f erenced 
tests as well, to better meet local needs. For example, as one program 
evaluator in an affluent Texas school district explained, in addition 
to administering the state mandated test, he also bought a* standardized / 
norm-referenced test for his district because f, the state assessment 
program was no,£*very helpful in comparing /our/ students to those in 
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the rest of t^ie nation/ 1 He added that after all, j students in his 
• district woul^L be competing against the national population for the 
^Is^st of their liv$s, so why compare them 'only to students in' Texas now? 

(Interview with Program Ev'aluator, 1/31/80, p. 1). An evaluator in 
* another state with its own assessment program reported that while the 

state test created lots of media interest "because districts across 
'^p/lifornia can compare themselves in reading and math, it is not very 
.valuable to teachers." He explained further that, 'as. a result, only * 

t _ w> 

a few even looked at the scores. His view was that standardized norm- 
referenced tests were much more helpful to teachers because of their 
analysis of individual students' achievement — ot lack of it— in the 



Same areas (Interview, 8/19/81, >p. 2), State-developed tests are 

unlikely to be as useful to locaf school districts, 

A related disadvantage of states developing their own tests is ' 

that ,these may be seen as a threat to local school district control. 

Through their testing program, states may be exercising more*" control 

over local curricula than local school districts want.* This may lead 
, to extended and complicated debate between state arid Ideal agencies 

overissues of 0 assessment. In some cases the validity of the state 

test has. already been challenged in the courts (e.g., Debra- P. v. 

Turlington iri Florida).; Local school districts ape far more receptive 
4 to tests produced by private testing f irms^because of th£ir political 



* Madaus \1979), for example, discusses this as a possible consequence 
of 'state minimum competency testing programs. 
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neutrality' anj* objectivity— despite apparent shortcomings in their 
products. 

Again, there, are some telling disadvantages in having state agencies 
take a more- active role in providing tests than under current market 
arrangements. One is that state agencies would not be able to provide 
much of a national base for comparing- student achievement. Another is 
that state-developed tests are likely to be designed to meet states' own 
needs, and would not easily allow for the exercise of greater political 
control by local school districts. In addition, state-developed tests 
might create another arena for conflict between local and state agencies, 
For these reasons, school districts are likely to want to purchase tests 

from private testing ^IrfflS- anyway. This appeara-to- be t-he-case In 

those states that have already developed their own tests. 

It is difficult, then, to assume that formally responsible govern- i 
ment agencies, at the local or state level, would better serve the needs 
.of school districts for educational assessment than does the marketplace. 
A further alternative is for 'the federal government to develop a test 
for use by local school districts, on either a mandatory or a voluntary 
"basis— as has already -been proposed in .Congressional hearings (Shoe- , 

i 

maker, 1978). The major advantage of this alternative is that it would 
offer a national base of comparison for assessing student achievement. 
But it suffers from a significant defect— one that became clear- during . 
the course of these hearings: the federal government's development of 
such a test might lead to an undue exercise of. control" over local 
school districts and be inconsistent with Constitutional delegation 



of authority over education to the state* This is indeed a major , 
problem, m 

Another possible role for the federal government, one more cTon- 
sistent with its Constitutional role, would be to regulate the develop- 
ment of national tests in the marketplace. This might be* done, as in 

other areas (e,g,, air travel)/ through some form of national regulatory 
/ * 

agency '(Hobbs, 1975), which would develop standards for tests and 
approve some tests over»other&. For examples it might provide some 
check on the curriculum basis and morning samples of the tests designed 
by private t est ing^f inns, -*lt might also provide consumer reports to 



disseminate its findings to the public, m 
- 1 — — School distr i ct political control ove r such an ageney-r^ewever T — — 

•would likely be problematic. The needs and preferences ot districts 

* * / v 

would differ and compete with one another for influence. The agency 
would theffefore be cotistra£ned in responding to any particular district 
and would likely became -a* lightning rod for many districts 1 criticisms 
and complaints. For political survival, then, it might have to confine 
its regulation of private testing firms to technical issues of test 
cons triict ion'. It would thus probably have no more power "and represent ^ 
school district .interests no better than ekisting private professional 
organizations like the .APA, or NCME, or AERA, that already issue tech- 

nical standards 'for test coftstruction, 

- - ^ »• • 

* In addition, a regulatory agency might be open to the -influence of 

'* » * . 

private testing firms acting in tfieir own behalf. There is considerable 
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evidence that this occurs in other areas of government regulation of 
private industry (McConnel,. 1967; Bauer, 1963). The rlsuTk might be the 
legitimization and ossification of the most powerful existing private 
^interests in the marketplace, as 'some have claimea has happened under 
other federal* agencies (e.g., the Civil Aeronautics Board /Hector, 1960/; 

the Food and Drug Administration /Hunter, 1972/; and the federal Com- 

i n * _ - 

munication Commission '/Friendly, 1959/). Thereby, private competition 

, 

might be restricted and its incentive to meet local district testing 
needs and preferences reduced. Conversely, the agency nfltght end up 
interfering with the activities of private testing firms and hinder 
their ability to respond to consumer needs. A common result of such 
r^ulatorj^genc ies— in-o£he*L_areas^ i t ional 



administrative costs for private industry, and in turn higher consumer 

costs (Scherer, 1971; Thurow, 1980). 1 

* 

SUMMARY AND CONCLUSIONS 

/ 

In summary, then,* while formal governance arrangements do offer 

/ - 

local school districts more political control over .phe development ,or 

■ • m r 

regulation of tests at the local, trie state, or the national level , 

4 w 
they 'also suffer from two major disadvantages compared to the market- 
place. One disadvantage is that the formally constituted role of 
governance of local, state, and federal agencies limits the extent to 
which they are legitimately responsible for developing national edu- . 

catlonal achievement tests. For example, because states and local 
» 

school districts are limited to thteir geographic areas, it is unlikely ^ 
that they could publicly justify the necessary t resources for the 
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development of a national test. The federal government does have some , 
responsibility for education at the national level. But since formal 
authority over education resides at the state and local levels, its 
national rc>le yould be constrained in either the development or the 
regulation of a national test. In contrast, private testing firms are 

not constrained -by formal political boundaries; their rolB as defined 

.<=* 

by the marketplace is national in scope. 

♦ 

' A second reason why local school, districts are unlikely to prefefc 

- formal governance arrangements is that they would invite political con- 
flict over the methodological and substantive issues of assessment. 
Local officials^ for example, may well find themselves involved in 
assessment issues at the local, state, and national level — hardly a 
welcome addition to the already, overwhelming demands placed upon them. * 
Again, this, problem is avoided in the marketplace where private testing 
firms are without formal political means of control or accountability; 
instead, they are dependent upon their consumers 1 ' informal perception 
of them as ,a neutral source of authority. 

/) Formal government .agencies, then, in assuming more, control, are 
, limited in the extent to which they can claim formal responsibility 
for developing a national test of educational achievement; and they 
run the risk of becoming embroiled, in political conflicts over the test~ 
development process. Furthermore, even if they were able to avoid these 
pitfalls, it is not clear that more control would necessarily lead to 
the development of a better test. There are inherent limitations in 
the ability of politics to produce the desired technical outcomes in 

■' . 48 
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the case of a national test of ^iuTcational achievement. For example, * 

** * ' » 

-the undue* influence of 'a group of school ^districts over the curriculum 

, <• * 

base of a state-developed or nationally Referenced test would likely 

compromise the standardization of its comparative base, and in turn the 

accuracy of its results . Greater political control by school districts 

may not, in fact, provide them with what they want. There may simply 

be an inverse relationship between maximum school district control and 

technically' better tests . * 

The limitations of politics for producing desired technical outcomes 

in the area of testing are perhaps analogous to, although not identical 

witfi, other areas-of formal government involvement in public policy * 

^areas. For example, while organized local, state, and national: public . 

interests may be able to influence the Environmental Protection Agency 

and its car pollution regulations, car manufacturers may simply be unable 

to produce a more pollution-free engine while providing the public with 

a desired level of autori^tive power. Similarly,, consumer interest" group 

pressure on the Food and Administration to take preservatives out , 

of food because' of possible harmful side effects may work against other 

consumers 1 preference for long-lasting food* As ip-the case , of testing, 

die ability to produce a product that wil^meet all consumer needs may 

be constrained by technical limitations—despite the opportunity for 

the political expression of these needs thorough formal governance 

arrangements. Government involvement may simply provide an arena for 

public and political debate while contributing little to, if not com- 

promising,' the meeting of public needs and wants through the marketplace. 
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In conclusion, then^it appears that the major strength of fdrmal 
governance alternatives W the markecplace--i,e. , providing local 
school districts with morp political control—may actyally' militate 

i * 

against the satisfaction of school districts' needs ^Private testing 
firms, operating through Ijhe marketplace without formal political 
control, seem better able to me£t those needs. This "is a furious 
conclusion in some ways, since it suggests that the exercd^T of 
school districts' political control through some form of formal 

governance arrangements is less effective than their exercise of con- 

i 

trel and influence through the marketplace. The marketplace, at least 
in the case! of achievement\testing, may proVide a degree of public con- 
trol tna£ formal public decision makin'g and politics cannot . 
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METHODOLOGY ' ' 

The purpose of this study was to explore whether and to what degree 
private testing firms (which publish norm-jreferenced educational achieve- 
ment tests) are held accountable by. local school- districts. It was 
assumed that local: districts 1 primary means of holding private testing 
firms accountable is through their role as consumers in the marketplace. 
With this assumption in mind, the study was designed to gain an under- 
standing of how a^id on what basis local schools decide to t>uy norm- 
referenced achievement tests from private testing firms. 
' Little^preVious work exists to guide the study. What related re- 
search is available focuses on particular aspects of the test selection 
process--e,g,., teacher involvement (Ward, 1980), or determination of 

the information needs of the test selectors. It also relies exclusively: 

X 

on survey methods and provides little in-depth analysis of Local test c 
selection. Furthermore, little, if any, attention has been paid to 
local school district views of and ^relationships witb private testing 
firms in .the selection process, A field-based case study method was " 



therefore adopted because of its established app«ij)riateness for pro- 
viding detailed and encompassing vfews of such uncharted areas of inquiry 
(Bogdan, 1975; Stake, 1977; 'Glazer § Strauss," 1973). What follows, is 
a description of the method, of this study. It is divided into thtee ' 
sections: site selection, conduct of field research, and recording and 
analysis of data* , I 



SITE SELECTION 

' w • 

In constructing the sample for. this -study, an initial effort 

e * 

was mad^ to select local school districts that were nationally repre- 
sentative -sf those who . buy norm-referenced achievement tests from 
private testing firms. Random selection procedures constitute the 
.traditional approach to defining any such sample. Under this appro^ch^ 
probability theory can they be used to generalize the results obtained 

to the universe from which the sample was drawn. However, rkndom 

* . / *■ 

sampling was not appropriate in this case, since the sample selected 

-A 

through this means would likely be considerably larger than could be 
accommodated by the resources of the study. Instead, an effort was 
made to -select local school districts according to the factors that 
may help explain their test selection decisions; Two such factors 
were identified through' preliminary interviews witdTpepple in the 
field: district size and state testing policies. (Other factors con r 
sidered included geographic location and per -pupil expenditures of 
local school districts; however, as there was little confirmation 
that these wete important, thej^were not used in site selection.) A 
third factor that was taken into account was the "ease of entry and 
access to school districts. The grounds for thj two substantive 
criteria are discussed below. 
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Selection by State 

9 

Several considerations went into "the selection of states in 
which' local s±tes were chosen. These considerations stemmed from 
the recognition, gained from both the literature and interviews 
with people in the field, ^hat^ certain state testing policies may 
influence local school districts K selection of tests from private 
testing .firms. While state t^'st selefction policies vary considerably 
across states, two policies in particular were irdentified^as important 
to include in the sample. One of these was state pplicy mandating , the 
use of a commercially published norm-referenced achievement test. 
California was selected because of its mandated use of a state- 
developed criterion-referenced test. And Massachusetts and Connecticut 
were ^elected because of the virtual absence of policy requiring a 
particular test to be used.* In each state, five local school sites , 
were vTsited; an extra pilot site was visited in Massachusetts. 



* Massachusetts has recently implemented a minimal competency testing 
program; but jit allows school districts^ to use either a state- 
developed criterion-referenced test on a commercially published 
norm-referenced achievement test.* % * ' < 

■ * i: : ■ [ 
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Selection by Sise 

Twenty-one local school' sites were visited for the study. In • 
.selecting these sites, an effort was made to represent school districts , 
of different size. To that end, three categories of school district 
size by student enrollment were developed from the seven standard cate- 
gories used by„the National Center for Educational Statistics; "large 
school districts," with enrollments of over 25,000; "medium-sized 
school districts," With enrollments of 4,000 to 24,999; and "small school 
districts," with enrollments of 4,000 Or less. Sites in each category 

< 

were then selected with two considerations in mind. One consideration 
was to, change sites'in proportion to the N.C.E.S. breakdown of local 
school districts by si2e. (This meant that 1.2% of the sample should 
be selected from large districts, 10.3% from medium-si2ed districts, and 
89.5% from small districts. _ But this consideration alone seemed J 
inadequate, since the t literature (National Association of School. Boards, 
1977;\Center for* the Study of Evaluation, 1978) and preliminary con- 
versations with people in the field suggested that only a low per- 
centage' o£ small school , districts bought commercially published nonft- * 
referenced achievement tests. Additionally ^ it was pointed out that 

the selection of such tests by* a larger schbgl district^ might* well 

* 

prove to be influential in the selection decisions of smaller districts. 
Therefore, it was deeded to increase th.e p r op o rtfton, *f sites selected 
from amon£ the medium and large districta. Hence 5 sites were ' 
Selected frqpf among^the large districts — representing 24% (approxi- 



mately) of the sample; 7 sites were selected from amoi^£\the medijum- 
sized districts — representing 33.5% (approximately) of the sample; 



SI 



and 9 sites were selected from among the Small districts — "representing 
43.5% (approximately) of the sample. f ; 

CONDUCT OF THE FIELD RESEARCH • # .• 

•The fcield research was conducted in three different settings: state 
educational agencies, private testing firms, and local school systems. 
However, the preponderance of the field work was conducted in local 
school districts, since the study was primarily concerned with their 
selection of tests.'? The field resear<5h in, each of ttyese settings is * 
discussed below, , • 

State Educational Agencies • 

In each of the states selected for the study, site visits were 
first made to staue departments of education. Relevant state officials 

r f * 

were interviewed, 4nd documents regarding state testing programs (or 

proposed programs) - and activities were gathered. Suggestions were also 

/ 

sought Regarding appropriate-llocal^site^ to visit. % Ten state department 
officials in all were contacted and interviewed. 

Private Testing Firms " • • 

Representatives o£ the major private testing ' firms were interviewed 
either personal ly " t>r on the phone throughout - the course of the f ield t 
research. Those interviewed "were selected because of their marketing 
background and/or direct contact, with local school districts v Inter- 
views were Held primarily as a .check on data gathered from school 
district officials as w*ll as to* explore particular Issues that emerged 
from visits to school districts* Six- such, interviews .were conducted* 
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whi£h typically confirmed what was reported by school district .of ficials. 
Documents of a technical and public relations nature were also gathered^ 

c • - 

from thes-e contacts with private testing firms. 

'Local School Systems * 

T^a^methods were vj^ed to gather' .jiata at the local school district ■ 

level — interviews and document review. , Of these, interviews with 

school personnel were the primary means . The format "for interviews was 

• * < * ' r 

semi-structured and open-ended and was guided by a protocol developed 

according to the issues under study. Interviews generally ranged from 

45 minutes to an hour, though some were as lotig as an hour and a half, 

while" others took only, one-half hciur.* 

Who was interviewed in each local s;chobl system depended upon who * 

was involved in test selection or was knowledgeable about it. This varied 

according to the nature and complexity of the test selection process in 

each system. Enough people were interviewed in each system to provide a 

clear and thorough understanding o£ that distfrict f s selection process. 

Typically, this involved several. interviews within each site, with' 

mAibers of the evaluation and testing department,, district administrators 

- and supervisors (generally in the math and English curriculum areas),, 

school principals, J guidance counselors, Jitle I' personnel, and teachers. 

The total number of interviews^ -conducted in local sites was 104 — 

approximately five interviews' per site. : 

In each site, several documents were reviewed and»analyzed. These 

e_j * » • 

documents primarily comprised internal school district memos, usually . 
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pertaining to the composition of school "dfstrict selection committees, 
and minutes or "Summaries of committee meetings. Documents reviewed also 
included information provided by private testing firms to local school 
districts. This info rmatioj^ ranged from technical booklets to public 
relations literature. Where relevant, other dociimeiits, such as the list 

' — 4 *- . " ft " 

of district curriculum objectives , *were also reviewed. Document review 
typically occurred on site so that school personnel involved *in' the 
selection" process could be identified and interviewed, and* issues 
pertinent to local decisions could be pursued "during the yisit. 

RECORDING ANt) ANALYSIS, OF DATA ' . 

• s 

Field Notes * * 

~ k . 

* All interviews and conversations, both formal and- informal, were 
recorded in field notes. Notes were taken by hand during interviews 
and conversations ,and later dictated for. transcription. They were dis-'< 
cursive in form and frequently wide-ranging. People were* often quoted 
verbatim arid the notes Became the researcher f s, personal record of wttet 
was saidr. An interview of one hour produced field notes yhich, when 
transcribed, had an average length of 6 to 8 single-spaced pages. In 

K * - ? . ■ • 4r , ~ 

addition to recording interviews, - tjife, fifeld researcher dictated summary 
notes" on visits" and notes .describing program *and district context. 



Close to "400 page^r «f\£ield notes were transcribed. 
Data Analysis 

_ ^ ^ , 

After field notes were transcribed, preliminary analyses were con- 
ducted and recorded in memos t These memos were submitted to the senio^ 
consultant fco^thj study for review. New areas of inquiry were suggested 
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and different ways of pursuing issues identified earlier were discussed. 
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*Ehis provided a basis for further developing and refining of the, inter- 

•4 

view protocol used in, the field work. A reiterative process of field worjc 
and analysis' was thereby established from the early stages of the study,/ 
Thus by the # end of the field work several memos representing progressive 
analysis of the data collected in the field were available for review/ 
These analyses providedJthe map for coding the field note data and for 
the final analysis. .One of the significant results of this process/ was * 
that, while considerable diversity in school district selection processes 
was , recorded, several eommon and general themes emerged whfen the data 
were analyzed across sites. Thus it was decided to 'present the findings 
for the study in > cross-site format rather than in individual cases. In 
writing the final report, a concerted effort was. thus made to select 
illustrative examples from the site£ visited that best illuminated these 
themes. , , 

Additional Data arid* Analysis 

• During the "course of the study, it became apparent that an NIE-funded 
research project concurrently being conducted,, by Huron on Ideal school 
jiistricts f use of evaluation and testing inforjife»t4on was ^collecting data 
relevant "to some o*f*the issues pttrsued in this study. Toward the end of 
the study; an .effort was therefore made to cull and review the relevant 



data fromf^the field^notes of that study. This allowed data from 18 addi- 
tional sites to be included in the final analysis. Where appropriate, 
illustrative examples from these sites were also used in preparation- of . 



the ^nal report. 
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